As video games become more social and acquire features permitted by increased computing power and high bandwidths, new ways to provide those games have evolved as well. An emerging game facilitation paradigm is referred to as cloud gaming services, e.g., the ONLIVE service. The concept of cloud gaming is to render 3D video games on cloud servers and then send each rendered game scene as a 2D video stream to the game player through broad-band networks.
The player's control signals (mouse, keyboard, or game controller events) are sent back to the cloud server to interact with the 3D video game. The cloud gaming service allows the gamer to play the most advanced 3D video games without buying high-end graphics hardware. In addition, cloud gaming also represents a new method of game distribution that can effectively prevent pirating. An exemplary cloud gaming topology 100 is illustrated schematically in FIG. 1, and includes a mobile client 101, rendering server 102, and mobile network(s) 103 connecting the two.
Of course, these benefits come at a price, albeit a modest one; cloud gaming depends on high bandwidth networks to deliver game video streams. For example, ONLIVE requires a wired network connection with no less than 5 Mbps constant bandwidth to provide 720 p 30 fps gaming services. Such a high-bandwidth requirement becomes a barrier for users who do not have broadband network connections. Thus, mobile users who have only mobile network connections cannot typically benefit from cloud gaming services.
Conventionally, within the cloud gaming paradigm, game frames are compressed with state-of-art H.264/AVC video encoders in real time to generate the video stream for users. However, compared to the general-purpose offline H.264/AVC encoder, the real-time H.264/AVC encoder cannot leverage optimizations that require a momentary look-ahead buffer or multiple encoding passes. Therefore, the real-time H.264/AVC encoder has a much lower encoding efficiency and leads to high-bandwidth requirements for cloud gaming services.
In U.S. Patent No. 2006/0088220A1 (Cheung et al.) a graphics to video encoder is presented that integrates both the base images rendered on the server and the client images constructed from client inputs and generate the video streams for the specified client user. However, the aforementioned reference focuses on the generation of customized client video stream but fails to address coding efficiency. In a related vein, US Patent No. 2010/0166068A1 (Perlman et al.) describes a system for cloud gaming services, and discusses reducing the overall system latency by smoothing the output bandwidth so that the latency caused by bandwidth peaks can be reduced, but it fails to reduce the overall bandwidth needed compared with the current H.264/AVC based video coder.
U.S. Pat. No. 7,194,033 B2 (Zhenya Alexander Yourlo) describes a technique that aims to efficiently encode images rendered by a graphical engine. However, the aforementioned method requires division of the input image frames into different categories based on the non-pixel changes between the current frame and the previous frame and applies different coding methods to different categories. The '033 invention also fails to apply graphics rendering contexts and image based rendering techniques to exploit frame coherence and assign the useful bits for the most important contents in the video. Moreover, it is not clear that the '033 method is available for real time video coding
U.S. Patent No. 2010/0166054 A1 (Wirick) describes a real-time video coding method that integrates online coding and off-line coding to improve coding performance. However, Wirick does not use graphics contexts to improve coding. Similarly, U.S. Patent No. 2011/0058021 A1 (Chen et al.) shares some concepts with the foregoing, but fails to apply 3D image warping to synthesize frames of different time. Moreover, the Chen invention fails to account for camera motion, and only the actually captured frames can be selected as reference frames. The same shortcomings are seen in U.S. Patent No. 2011/0069152 (Wang et al.) and U.S. Patent No. 2010/0329358 A1 (Zhang et al.).